cPanel Forums

Cloudflare is a heavy user of Kubeetes for engineering workloads: it's used to power the backend of our APIs, to handle batch-processing such as analytics aggregation and bot detection, and engineering tools such as our CI/CD pipelines. But between load balancers, API servers, etcd, ingresses, and pods, the surface area exposed by Kubeetes can be rather large.In this post, we share a little bit about how our engineering team dogfoods Cloudflare Zero Trust to secure Kubeetes — and enables kubectl without proxies.Our General Approach to Kubeetes SecurityAs part of our security measures, we heavily limit what can access our clusters over the network. Where a network service is exposed, we add additional protections, such as requiring Cloudflare Access authentication or Mutual TLS (or both) to access ingress resources.These network restrictions include access to the cluster's API server. Without access to this, engineers at Cloudflare would not be able to use tools like kubectl to introspect their team's resources. While we believe Continuous Deployments and GitOps are best practices, allowing developers to use the Kubeetes API aids in troubleshooting and increasing developer velocity. Not having access would have been a deal breaker.To satisfy our security requirements, we're using Cloudflare Zero Trust, and we wanted to share how we're using it, and the process that brought us here.Before Zero TrustIn the world before Zero Trust, engineers could access the Kubeetes

Throughout Cloudflare One week, we provided playbooks on how to replace your legacy appliances with Zero Trust services. Using our own products is part of our team’s culture, and we want to share our experiences when we implemented Zero Trust. Our jouey was similar to many of our customers. Not only did we want better security solutions, but the tools we were using made our work more difficult than it needed to be. This started with just a search for an alteative to remotely connecting on a clunky VPN, but soon we were deploying Zero Trust solutions to protect our employees’ web browsing and email. Next, we are looking forward to upgrading our SaaS security with our new CASB product.We know that getting started with Zero Trust can seem daunting, so we hope that you can lea from our own jouey and see how it benefited us.Replacing a VPN: launching Cloudflare AccessBack in 2015, all of Cloudflare’s inteally-hosted applications were reached via a hardware-based VPN. On-call engineers would fire up a client on their laptop, connect to the VPN, and log on to Grafana. This process was frustrating and slow. Many of the products we build are a direct result of the challenges our own team is facing, and Access is a perfect example. Launching as an inteal project in 2015, Access enabled employees to access inteal applications through our identity provider. We started with just one application behind Access with the goal of improving incident response times. Engineers who r

Zscaler has been building out its security offerings for 15 years. Cloudflare is 13 years old, and we have been delivering Zero Trust for the last four. This sounds like we are a late starter — but in this post, we’re going to show that on total Zero Trust, SSE, SASE and beyond, Cloudflare One functionality surpasses that of Zscaler Zero Trust Exchange. .tg {border-collapse:collapse;border-spacing:0;} .tg td{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px; overflow:hidden;padding:10px 5px;word-break:normal;} .tg th{border-color:black;border-style:solid;border-width:1px;font-family:Arial, sans-serif;font-size:14px; font-weight:normal;overflow:hidden;padding:10px 5px;word-break:normal;} .tg .tg-onfx{background-color:#afe4fd;color:#3663ae;font-size:18px;font-weight:bold;text-align:center;vertical-align:top} .tg .tg-baqh{text-align:center;vertical-align:top} .tg .tg-c3ow{border-color:inherit;text-align:center;vertical-align:top} .tg .tg-cmei{background-color:#afe4fd;border-color:inherit;color:#3663ae;font-size:18px;font-weight:bold;text-align:left; vertical-align:top} .tg .tg-8d3w{background-color:#FFF;border-color:inherit;color:#222;font-weight:bold;text-align:center;vertical-align:top} .tg .tg-q8v8{background-color:#afe4fd;border-color:inherit;color:#3663ae;font-size:18px;font-weight:bold;text-align:center; vertical-align:top} .tg .tg-0pky{border-color:inherit;text-align:left;vertical-align:top} .tg .tg-gct1{backgro

If you’ve tuned into this blog for long enough, you’ll notice that we’re pretty big on using and stress-testing our own products (“dogfooding”) at Cloudflare.That applies to our security team, product teams, and – as my colleague Kristian just blogged about – even our documentation team. We’re incredibly excited to be on the Pages platform, both because of the performance and workflow improvements and the opportunity to help the platform develop.What you probably haven’t heard about is how our docs team uses dogfooding – and data – to improve our documentation.Dogfooding for docsAs a technical writer, it’s pretty common to do the thing you’re documenting. After all, it’s really hard to write step-by-step instructions if you haven’t been through those steps. It’s also a great opportunity to provide feedback to our product teams.What’s not as common for a writer, however, is actually using the thing you’re documenting. And it’s totally understandable why. You’re already accountable to your deadlines and product managers, so you might not have the time. You might not have the technical background. And then there’s the whole problem of a real-world use case. If you’re really dedicated, you can set up a personal project… but it's hard to replicate real-world situations and even harder to simulate real-world motivation.And that brings me to one of the coolest parts of our docs team. We actually manage the Cloudflare settings for our docs website, developers.cloudflare.com. There’s

In September 2021, we shared extensive benchmarking results of 1,000 networks all around the world. The results showed that on a range of tests (TCP connection time, time to first byte, time to last byte), and on different measures (p95, mean), Cloudflare was the fastest provider in 49% of the top 1,000 networks around the world.Since then, we’ve expanded our testing to cover not just 1,000 but 3,000 networks, and we’ve worked to continuously improve performance, with the ultimate goal of being the fastest everywhere and an intermediate goal to grow the number of networks where we’re the fastest by at least 10% every Innovation Week. We met that goal Platform Week May 2022), and we’re carrying the work over to Cloudflare One Week (June 2022).We’re excited to share that Cloudflare was the fastest provider in 1,290 of the top 3,000 most reported networks, up from 1,280 even one month ago during Platform Week.Measuring what mattersTo quantify global network performance, we have to get enough data from around the world, across all manner of different networks, comparing ourselves with other providers. We use Real User Measurements (RUM) to fetch a 100kB file from different providers. Users around the world report the performance of different providers.The more users who report the data, the higher fidelity the signal is. The goal is to provide an accurate picture of where different providers are faster, and more importantly, where Cloudflare can improve. You can read more about t

A fundamental principle here at Cloudflare has always been that we want to serve everyone - from individual developers to small businesses to large corporations. In the earliest days, we provided services to hosting partners and resellers around the globe, who helped bring Cloudflare to thousands of domains with free caching and DDoS protection for shared infrastructures.Today, we want to reinforce our commitment to our hosting ecosystem and small business partners that leverage Cloudflare to help bring a better Inteet experience to their customers. We've been building a robust multi-tenant partner platform that we will begin to open up to everyone searching for a faster, safer, and better Inteet experience. This platform will come in the form of a Self Serve Partner program that will allow SMB agencies & hosting partners to create accounts for all their customers under one dashboard, consolidate billing, and provide discounted plans to our partners.Deprecation of our legacy APIsTo make way for the new, we first must discuss the end-of-life of some of Cloudflare’s earliest APIs. Built and launched in 2011, our Hosting and Optimized Partner Programs allowed our initial CDN and DDoS solutions to expand to brand-new audiences around the globe. These APIs were essential for fueling growth in the earliest days of Cloudflare supporting reseller partners, hosting partners, and exteal plugins that helped make implementing Cloudflare easier than ever. On November 1, 2022 - C

You may have heard a bit about the Hertzbleed attack that was recently disclosed. Fortunately, one of the student researchers who was part of the team that discovered this vulnerability and developed the attack is spending this summer with Cloudflare Research and can help us understand it better.The first thing to note is that Hertzbleed is a new type of side-channel attack that relies on changes in CPU frequency. Hertzbleed is a real, and practical, threat to the security of cryptographic software.Should I be worried?From the Hertzbleed website,“If you are an ordinary user and not a cryptography engineer, probably not: you don’t need to apply a patch or change any configurations right now. If you are a cryptography engineer, read on. Also, if you are running a SIKE decapsulation server, make sure to deploy the mitigation described below.”Notice: As of today, there is no known attack that uses Hertzbleed to target conventional and standardized cryptography, such as the encryption used in Cloudflare products and services. Having said that, let’s get into the details of processor frequency scaling to understand the core of this vulnerability.In short, the Hertzbleed attack shows that, under certain circumstances, dynamic voltage and frequency scaling (DVFS), a power management scheme of mode x86 processors, depends on the data being processed. This means that on mode processors, the same program can run at different CPU frequencies (and therefore take different wall-clock t

Linux Security Modules (LSM) is a hook-based framework for implementing security policies and Mandatory Access Control in the Linux keel. Until recently users looking to implement a security policy had just two options. Configure an existing LSM module such as AppArmor or SELinux, or write a custom keel module.Linux 5.7 introduced a third way: LSM extended Berkeley Packet Filters (eBPF) (LSM BPF for short). LSM BPF allows developers to write granular policies without configuration or loading a keel module. LSM BPF programs are verified on load, and then executed when an LSM hook is reached in a call path.Let’s solve a real-world problemMode operating systems provide facilities allowing "partitioning" of keel resources. For example FreeBSD has "jails", Solaris has "zones". Linux is different - it provides a set of seemingly independent facilities each allowing isolation of a specific resource. These are called "namespaces" and have been growing in the keel for years. They are the base of popular tools like Docker, lxc or firejail. Many of the namespaces are uncontroversial, like the UTS namespace which allows the host system to hide its hostname and time. Others are complex but straightforward - NET and NS (mount) namespaces are known to be hard to wrap your head around. Finally, there is this very special very curious USER namespace.USER namespace is special, since it allows the owner to operate as "root" inside it. How it works is beyond the scope of this blog po

On May 19, 2021, a Microsoft blog post announced that “The future of Inteet Explorer on Windows 10 is in Microsoft Edge” and that “the Inteet Explorer 11 desktop application will be retired and go out of support on June 15, 2022, for certain versions of Windows 10.” According to an associated FAQ page, those “certain versions” include Windows 10 client SKUs and Windows 10 IoT. According to data from Statcounter, Windows 10 currently accounts for over 70% of desktop Windows market share on a global basis, so this “retirement” impacts a significant number of Windows systems around the world.As the retirement date for Inteet Explorer 11 has recently passed, we wanted to explore several related usage trends:Is there a visible indication that use is declining in preparation for its retirement?Where is Inteet Explorer 11 still in the heaviest use?How does the use of Inteet Explorer 11 compare to previous versions?How much Inteet Explorer traffic is “likely human” vs. “likely automated”?How do Inteet Explorer usage pattes compare with those of Microsoft Edge, its replacement?The long goodbyePublicly released in January 2020, and automatically rolled out to Windows users starting in June 2020, Chromium-based Microsoft Edge has become the default browser for the Windows platform, intended to replace Inteet Explorer. Given the two-year runway, and Microsoft’s May 2021 announcement, we would expect to see Inteet Explorer traffic decline over time as users shift to Ed

Managed Transforms is the next step on a jouey to make HTTP header modification a trivial task for our customers. In early 2021 the only way for Cloudflare customers to modify HTTP headers was by writing a Cloudflare Worker. We heard from numerous customers who wanted a simpler way.In June 2021 we introduced Transform Rules, giving customers a simple UI letting them specify what the custom HTTP header’s name and value is—either a static string (i.e. X-My-CDN: Cloudflare) or a dynamically populated value (i.e. X-Bot-Score: cf.bot_management.score).This made the job much simpler, however there is still a good amount of thought required—with a number of potential drop-off points on the user jouey. For example, in order to dynamically populate the bot score into the value of an HTTP request header, the user needs to know the correct field name. To find that they'll need to go to the documentation site, find the correct section, etc.When we analyzed how our customers use Transform Rules we found a set of very common use cases in the data. Four of the top eight fields used were relating to bot management; customers wanting to have the bot score, JA3 hash, etc. of each request added as an HTTP header. A further three of the top 10 fields were relating to the geographic location of the visitor (their city, country and ASN). We also saw over 400 Transform Rules being used just to remove X-Powered-By. That means potentially 400 users went to the same part of the dashboard, typed th

Here at Cloudflare we're constantly working on improving our service. Our engineers are looking at hundreds of parameters of our traffic, making sure that we get better all the time.One of the core numbers we keep a close eye on is HTTP request latency, which is important for many of our products. We regard latency spikes as bugs to be fixed. One example is the 2017 story of "Why does one NGINX worker take all the load?", where we optimized our TCP Accept queues to improve overall latency of TCP sockets waiting for accept().Performance tuning is a holistic endeavor, and we monitor and continuously improve a range of other performance metrics as well, including throughput. Sometimes, tradeoffs have to be made. Such a case occurred in 2015, when a latency spike was discovered in our processing of HTTP requests. The solution at the time was to set tcp_rmem to 4 MiB, which minimizes the amount of time the keel spends on TCP collapse processing. It was this collapse processing that was causing the latency spikes. Later in this post we discuss TCP collapse processing in more detail.The tradeoff is that using a low value for tcp_rmem limits TCP throughput over high latency links. The following graph shows the maximum throughput as a function of network latency for a window size of 2 MiB. Note that the 2 MiB corresponds to a tcp_rmem value of 4 MiB due to the tcp_adv_win_scale setting in effect at the time.For the Cloudflare products then in existence, this was not a major problem,

Here’s a short list of recent technical blog posts to give you something to read today.Inteet Explorer, we hardly knew yeMicrosoft has announced the end-of-life for the venerable Inteet Explorer browser. Here we take a look at the demise of IE and the rise of the Edge browser. And we investigate how many bots on the Inteet continue to impersonate Inteet Explorer versions that have long since been replaced.Live-patching security vulnerabilities inside the Linux keel with eBPF Linux Security ModuleLooking for something with a lot of technical detail? Look no further than this blog about live-patching the Linux keel using eBPF. Code, Makefiles and more within!Hertzbleed explainedFeeling mathematical? Or just need a dose of CPU-level antics? Look no further than this deep explainer about how CPU frequency scaling leads to a nasty side channel affecting cryptographic algorithms.Early Hints update: How Cloudflare, Google, and Shopify are working together to build a faster Inteet for everyoneThe HTTP standard for Early Hints shows a lot of promise. How much? In this blog post, we dig into data about Early Hints in the real world and show how much faster the web is with it.Private Access Tokens: eliminating CAPTCHAs on iPhones and Macs with open standardsDislike CAPTCHAs? Yes, us too. As part of our program to eliminate captures there’s a new standard: Private Access Tokens. This blog shows how they work and how they can be used to prove you’re human without saying who

Last year during CIO week, we announced Page Shield in general availability. Today, we talk about improvements we’ve made to help Page Shield users focus on the highest impact scripts and get more value out of the product. In this post we go over improvements to script status, metadata and categorization.What is Page Shield?Page Shield protects website owners and visitors against malicious 3rd party JavaScript. JavaScript can be leveraged in a number of malicious ways: browser-side crypto mining, data exfiltration and malware injection to mention a few.For example a single hijacked JavaScript can expose millions of user’s credit card details across a range of websites to a malicious actor. The bad actor would scrape details by leveraging a compromised JavaScript library, skimming inputs to a form and exfiltrating this to a 3rd party endpoint under their control.Today Page Shield partially relies on Content Security Policies (CSP), a browser native framework that can be used to control and gain visibility of which scripts are allowed to load on pages (while also reporting on any violations). We use these violation reports to provide detailed information in the Cloudflare dashboard regarding scripts being loaded by end-user browsers.Page Shield users, via the dashboard, can see which scripts are active on their website and on which pages. Users can be alerted in case a script performs malicious activity, and monitor code changes of the script.Script statusTo help identify malic

Cloudflare operates in more than 270 cities in over 100 countries, where we interconnect with over 10,000 network providers in order to provide a broad range of services to millions of customers. The breadth of both our network and our customer base provides us with a unique perspective on Inteet resilience, enabling us to observe the impact of Inteet disruptions. In many cases, these disruptions can be attributed to a physical event, while in other cases, they are due to an intentional govement-directed shutdown. In this post, we review selected Inteet disruptions observed by Cloudflare during the second quarter of 2022, supported by traffic graphs from Cloudflare Radar and other inteal Cloudflare tools, and grouped by associated cause or common geography.Optic outagesThis quarter, we saw the usual complement of damage to both terrestrial and submarine fiber-optic cables, including one that impacted multiple countries across thousands of miles, and another more localized outage that was due to an errant rodent.ComcastOn April 25, Comcast subscribers in nearly 20 southweste Florida cities experienced an outage, reportedly due to a fiber cut. The traffic impact of this cut is clearly visible in the graph below, with Cloudflare traffic for these cities dropping to zero between 1915–2050 UTC (1515–1850 local time).Not only did the fiber cut force a significant number of Comcast subscribers offline, but it also impacted the types of traffic observed across Comcast as

Welcome to our 2022 Q2 DDoS report. This report includes insights and trends about the DDoS threat landscape — as observed across the global Cloudflare network. An interactive version of this report is also available on Radar.In Q2, we’ve seen some of the largest attacks the world has ever seen including a 26 million request per second HTTPS DDoS attacks that Cloudflare automatically detected and mitigated. Furthermore, attacks against Ukraine and Russia continue, whilst a new Ransom DDoS attack campaign emerged.The HighlightsUkrainian and Russian InteetThe war on the ground is accompanied by attacks targeting the spread of information.Broadcast Media companies in the Ukraine were the most targeted in Q2 by DDoS attacks. In fact, all the top five most attacked industries are all in online/Inteet media, publishing, and broadcasting.In Russia on the other hand, Online Media drops as the most attacked industry to the third place. Making their way to the top, Banking, Financial Services and Insurance (BFSI) companies in Russia were the most targeted in Q2; almost 45% of all application-layer DDoS attacks targeted the BFSI sector. Cryptocurrency companies in Russia were the second most attacked.Read more about what Cloudflare is doing to keep the Open Inteet flowing into Russia and keep attacks from getting out.Ransom DDoS attacksWe’ve seen a new wave of Ransom DDoS attacks by entities claiming to be the Fancy Lazarus.In June 2022, ransom attacks peaked to the highest of the